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Ultra-Large-Scale  (ULS) 
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Increasing  Scale  In  Military  Systems 


Increasingly  Complex  Systems 

•  ultra-large,  network-centric,  real-time, 
cyber-physical-social  systems 

—  thousands  of  platforms,  sensors, 
decision  nodes,  weapons,  and 
warfighters 

—  connected  through  heterogeneous 
wired  and  wireless  networks 

•  Transient  and  enduring  resource 
constraints  and  failures 

•  Continuous  adaptation 

•  Sustainable  -  legally,  technically, 
politically 
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Ultra-Large-Scale  (ULS)  Systems  Study 


Asst  Sec  Army 
Claude  Bolton 

August  16,  2005 


« Gather  leading  experts  to 

study  these  ULS  systems  of 
the  future. 


“...How can  future 
systems,  which  are 
likely  to  be  a  billion 
lines  of  code,  be 
built  reliably  if  we 
can’t  even  get 
today’s  systems 
right?” 


Intended  outcomes: 

•ULS  System  Research 
Agenda 

•program  proposal 

•collaborative  research 
network 


About  the  Effort 

Funded  by  the  Army  (ASA  ALT) 

Created  and  led  by  the  SEI 

Staffing:  9  member  SEI  team 

13  member  expert  panel 


Duration:  one  year  (04/05  -  05/06) 
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ULS  Systems  Research  Study  Report 


http://www.sei.cmu.edu/uls/ 
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ULS  Systems  Research  Agenda 


Describes 

•  the  characteristics  of  ULS 
systems 

•  the  associated  challenges 

•  promising  research  areas 
and  topics 

Is  based  on  a  new  perspective 
needed  to  address  the 
problems  associated  with 
ultra-large-scale  systems. 
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What  Is  an  Ultra-Large-Scale  (ULS)  System? 


A  ULS  System  has  unprecedented  scale  in  some  of  these  dimensions: 

•  lines  of  code 

•  amount  of  data  stored,  accessed,  manipulated,  and  refined 

•  number  of  connections  and  interdependencies 

•  number  of  hardware  elements 

•  number  of  computational  elements 

•  number  of  system  purposes  and  user  perception  of  these  purposes 

•  number  of  routine  processes,  interactions,  and  “emergent  behaviors” 

•  number  of  (overlapping)  policy  domains  and  enforceable  mechanisms 

•  number  of  people  involved  in  some  way 


ULS  systems  are  interdependent  webs  of  software-reliant  systems, 
people,  policies,  cultures,  and  economics. 

V  J 
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Consequences  of  Scale 


Characteristics  of  ULS  systems  arise  because  of  their  scale. 

•  Decentralization 

•  Inherently  conflicting,  unknowable,  and  diverse 
requirements 

•  Continuous  evolution  and  deployment 

•  Heterogeneous,  inconsistent,  and  changing  elements 

•  Erosion  of  the  people/system  boundary 

•  Normal  failures 

•  New  paradigms  for  acquisition  and  policy 
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Approaches  to  Software  Development 


The  Engineering  Perspective 


AGILE 

SOFTWARE  DEVELOPMENT 

MANIFESTO 

INDIVIDUALS  AND  INTERACTIONS 

OVER  PROCESSES  AND  TOOLS 

WORKING  SOFTWARE 

OVER  COMPREHENSIVE  DOCUMENTATION 

CUSTOMER  COLLABORATION 

OVER  CONTRACT  NEGOTIATION 

RESPONDING  TO  CHANGE 

OVER  FOLLOWING  A  PLAN 


The  Agile  Perspective 
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A  New  Perspective  is  Required 


“The  older  is  not  always  a  reliable  model  for  the  newer,  the 
smaller  for  the  larger,  or  the  simpler  for  the  more 
complex... Making  something  greater  than  any  existing  thing 

necessarily  involves  going  beyond  experience.” 

Henry  Petroski 

Pushing  the  Limits:  New  Adventures  in  Engineering 
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Think  Cities  not  Buildings 


“Cities  are  places  of  massive 
information  flows,  networks,  and 
conduits,  and  myriad  transitory 
information  exchanges.” 
Howard  Rheinegold:  Smart  Mobs 
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Think  Ecosystem 
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Diverse  users  with  complex  networked 
dependencies  and  intrinsic  adaptive 
behavior 

Has: 

•  Robustness  mechanisms:  achieving 
stability  in  the  presence  of  disruption 

•  Measures  of  health:  diversity, 
population  trends,  other  key  indicators 
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Think  Socio-Technical  Ecosystems 


Socio-technical  ecosystems 

include  people,  organizations,  and 
technologies  at  all  levels  with 
significant  and  often  competing 
interdependencies. 

•  dynamic  communities 

•  interaction  between  and 
among  all  entities  -  roles, 
responsibilities,  and 
information  flows 

•  competition  for  resources 

•  rules,  incentives,  and 
adaptation 
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Challenges 


ULS  systems  will  present  challenges 
in  three  broad  areas: 

•  Design  and  evolution 

•  Orchestration  and  control 

•  Monitoring  and  assessment 


“There  are  challenges  associated 
with  ULS  systems  that  today’s 
perspectives  are  very  unlikely  to 
be  able  to  address.” 

V _ y 
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Research  Portfolio 


6.1  Human  Interaction 

6.2  Computational  Emergence 

6.3  Design 

6.4  Computational  Engineering 

6.5  Adaptive  System  Infrastructure 

6.6  Adaptable  and  Predictable  System  Quality 

6.7  Policy,  Acquisition,  and  Management 
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What  We  Learned 


There  is  an  unstoppable  trend  toward  increasing  scale  in 
many  systems  important  to  our  society. 

Scale  changes  everything. 

These  changes  undermine  the  assumptions  we  routinely 
make  in  traditional  software  engineering  approaches. 

Manifestations  of  scale  and  its  attendant  complexity  arise 
in  many  disciplines,  and  can  be  understood  as  a 
phenomenon  in  its  own  right. 

New,  interdisciplinary  perspective  and  new  research  in 
building  ultra-large-scale  systems  is  long  overdue. 
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Our  Assertion 


“Fundamental  gaps  in  our  current 
understanding  of  software  and  its 
development  at  the  scale  of  ULS 
systems  present  profound  impediments 
to  the  achievement  of  mission 
objectives.  These  gaps  are  strategic, 
not  tactical.  They  are  unlikely  to  be 
addressed  by  incremental  research  in 
established  categories.  We  require  a 
broad  new  conception  of  both  the 
nature  of  such  systems  and  new  ideas 
for  how  to  develop  them.” 
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Early  Post-Study  Observations 


•  We  never  suggested  that  all  systems  of  the  future  will  be 
ULS  systems.  Clearly,  they  won’t  be. 

•  What  you  call  it  (system  of  systems,  ULS  system,  complex 
net-centric  system)  is  really  unimportant. 

•  It  is  important  that  ULS  system  characteristics  are 
recognized. 

•  Systems  engineering  does  not  have  all  the  answers. 

•  Not  having  a  research  area  on  network  security  was  a 
lightening  rod. 

•  The  research  identified  in  the  ULS  system  study  has  a 
positive  impact  on  systems  that  are  not  ULS. 


Seven  Years  Later 
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Since  Then 
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Societal  Problems 


Climate  change  and  the  environment 
Powerinc 

\ii^ 

Disease,  epidemics,  and  health  care 
Livabie  megacities 
Safety  and  security 
Transportation 


itC/O 
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Carnegie  Mellon 
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Software  is  Ubiquitous  and  Often  Transparent 
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Software-Reliant  Systems:  What  HAS  Changed? 
Increased  connectivity 

«e«Kii  JF 

Challenges 

•  scale  and  complexity 

•  decentralization  and  distribution 

□ 

•  “big  data” 

•  increased  operational  tempo 

•  mismatched  ecosystem  tempos 

•  vulnerability  ; 

•  collective  action 

•  disruptive  and  emerging  technologies 


□ 


Software  Engineering  Institute  Can 
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Our  Milieu 


HEADLINES 


CH 


CO 


SK 


FYFON 


'^CYBERSECURITY  STARTS  IN  HIGH  SCHOOL  WITH  TOMORROWS  HIRES 

CISPA  CYBERSECURITY  BILL  BACKERS  HOPE  SECOND  TIME'S  A 

NHAJ  CAN  BE  DONE  ABOUT  THE  CYBEBSECURITY  SKILLS  SHORTAGE? 

WHAT  COMES  AFTER  THE  CYBER  SECURITY  STRATEGY? 

GOVERNMENT  TO  SHARE  CYBER  SECURITY  INFORMATION 

rOP  us  ADMIRAL  PUTS  CYBER  SECURITY  ON  THE  N 

JS  INVESTMENT  IN  CYBER-SECURITY  EQUAL  TO  NUCL 

GLOBAL  CYBER  DEFENSE  EXPERTS  TAKE  ON  H 

m  FORCE  HACKERS  WIN  NSA'S  13TH  ANNUAL 

CURTIS  LEVINSON,  UNITED  STATES  CYB 

JAPAN,  U.S.  CONCLUDE  1ST  DIALOGUE 

FIOCHESTER  INSTITUTE  OF  TECHN 

GOVERNMENT  TO  SHARE  CYBER^ 

INCREASE  COOPERATION  T 

VJEW  SOFTWARE  SPOTS, 

CARS’  INTERNET 

(ERALA  GETS 

HACKING 
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The  “Crowd” 
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GitHub 

match.com 


DARPA  BAA  11-64:  Social  Media  in 
Strategic  Communication  (SMI SC) 


Research  to  investigate  innovative  approaches  that 
enabie  revolutionary  advances  in  science,  devices,  or 
systems  for  strategies  to: _ 


1.  Detect,  classify  formation, 

development  andi^ffi^Wlff^i^nBl!?iriWjoafmemes^.  and 
(b)  purposeful  or  deceptive  messaging  and  misinformation. 

infiuence 

operations  across  social  media  sites  and  communities. 


3.  Identify  participants  and  intent,  and  measure  effects  of 
persuasion  campaigns. 

4.1 

operations? 


Counter  messaging 


of  detected  adversary  influence 
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Some  ULS  Systems  Buzz 


-21,266  downloads  and  hardcopy 

75+  citations  in  refereed  publications 

Presentations 

Workshops 

Blog,  journal,  and  twitter  references 
Initiatives  and  degree  programs 


LSCITS 


Health  IT  as  an  Ultra  Large-Scale  System 


Dr.  Doug  Fridsma 


Chief  Science  Office  and  Director 
Office  of  Science  and  Technology 
Office  of  National  Coordinator 
US  Health  and  Human  Services 


Health  IT  Buzz 

February  21, 2013 

http://www.healthit.gov/buzz-blog/electronic- 
health-and-medical-records/healthcare- 
\^building-interoperable-health-system-tough/  ^ 
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More  Buzz 


Ultralarge  Systems: 

Redefining  Software 
Engineering? 

Greg  Goth 

March/April  2008  IEEE  Software 


although  the  ULSS  report  focused  on 
challenges  faced  by  the  United  States 
Department  of  Defense  in  engineering 
software  intensive  systems,  “its  description 
of  how  the  fundamental  principles  of 
software  design  will  change  in  a  global 
economy  ...  is  finding  wide  appeal.” 


Managing  Scale  and  Agility: 


A 


Transformational  Architecture  for 
the  Smart  Grid 


Wayne  Longcore 


SATURN  2010 


“We  are  creating  the  first  true  instantiation  of 
a  high-functioning  Ultra-Large-Scale 
System — the  Smart  Grid.” 


Notes  on  ultra-large-scale  systems 


http://blogJohnrooksby.  org/post/1 3296 
33/notes-on-ultra-large-scale-systems 

John  Rooksby 

University  of  Glasgow 


“True  to  national  stereotypes  the  Americans 
were  asking  how  can  we  build  the  biggest 
systems  in  the  world?  The  British  were 
asking  how  can  we  stop  screwing  up  when 
we  try  to  build  the  biggest  systems  in  the 
world?" 

— ^ 
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Upon  Reflection 


ULS  Systems 
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ULS  System  Perspective 
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Selected  Experiences  with 
Systems  at  Scale: 

Nibbling  at  the  Edges 
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Department  of  Energy:  Smart  Grid 


Extreme  Science  and  Engineering 
Discovery  Environment 


The  extreme  Science  and 
Engineering  Discovery 
Environment  (XSEDE)  enhances 
the  productivity  of  scientists 
and  engineers. 


/  Orflantxaltgin^;  Ptf>cy.  Buwtpm  ObjgelwqB.  Prooedtf  b 
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Diagrams  courtesy  of 
Wayne  Longcore 
Consumers  Energy 
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Smart  Grid  -  A  ULS  System 


GoneeffwaiMatlei-lRemasaTeo! 


Diagrams  courtesy  of 
Wayne  Longcore 
Consumers  Energy 


imeiama^lity  layers  a  thetllotlel 


/  Qraaniatitional:  Policy  ,  Buaifiasa  ObiQClitf»£.  Business  Procedures 
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Specific  Probiem  and  Technicai  Approach 

Problem 


Create  a  capability  to  discover  if  an  intruder  is  executing  foreign  code  in  the 
systems  running  US  critical  infrastructure  (e.g.,  Stuxnet). 

Approach 


Exploit  known  performance  characteristics  of  critical  devices  (timing 
profiies)  and  monitor  run-time  behavior  for  deviations. 


Intelligent  Electronic  Devices  (devices  deployed  to  control  field  equipment) 
exhibit  several  desirable  characteristics 

•  are  real-time  systems 

•  are  deployed  in  known,  stable  configurations 

•  react  to  a  reasonably  small  number  of  kinds  of  stimulus 


Key  ^  job  start 
^  job  end 

I  I  job  execution 


A  timing  violation  occurs  when 

•  job  execution  is  too  short  or  too  long 

•  job  release  period  is  too  short  or  too 
long 


MustWait  P 


n 


CanWait_P 

- >1 


1  MustWait  C  1 

1  CanWait  C 

k - — — w— 

— k— — - - 

time 
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Broader  ULS  System  Impact 


The  expansion  of  communication  among  diverse  devices  being  seen  in 
the  Smart  Grid  is  also  happening  in  other  ULS  systems  and  raises  the 
same  concerns  for  a  capability  to  detect  this  class  of  intrusions. 


Other  real-time  systems  with  knowable  timing  profiles  where  the 
technique  could  be  used  to  enhance  intrusion  detection  include 

•  sensors 

•  fire  control  systems 

•  vehicle  and  engine  controllers 

•  avionics  systems 
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Extreme  Science  and  Engineering 
Discovery  Environment 


The  extreme  Science  and 
Engineering  Discovery 
Environment  (XSEDE)  enhances 
the  productivity  of  scientists  and 
engineers. 


Ejdffifne  5d6nc£  and  EngineBriiig 


XSEDE  is  the  framework  for  a  national 
cyber-infrastructure  ecosystem,  serving  as 
a  platform  for  multiscale  cyber¬ 
infrastructure  integration  for  scientific 
collaboration. 
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Extreme  Science  and  Engineering 
Discovery  Environment 


XSEDE’s  innovative,  open  standards- 
based  architecture  facilitates  an 
unparalleled  level  of  integration. 


Enabling  this  architecture  are 
XSEDE’s  professional  systems 
engineering  approach  and 
technology  insertion  efforts,  which 
ensure  robustness  and  security 
while  continuously  incorporating 
new  technologies. 
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Extreme  Science  and  Engineering 
Discovery  Environment 


Technical  problem:  XSEDE  needs  well-defined  software  development 
and  software  management  practices  across  the  XSEDE  partner  network 
before  it  can  embrace  practices  more  appropriate  for  a  socio-technical 
ecosystem. 

Approach: 

•  Identify  and  coach  XSEDE  community  in  adopting  a  variant  of 
architecture-centric  practices  suitable  for  their  collaborative  ecosystem 

•  Apply  automated  text  and  social  network  analyses  to  data  gathered 
from  XSEDE  with  the  goal  of  providing  automated  infrastructure  support 
for  ensuring  that  the  right  people  get  the  right  information  at  the  right 
time. 

Impact:  Enabling  the  evolution  of  the  nation’s  scientific  computing  grid  via 
architecture-centered  practices  and  serving  as  an  exemplar  for  socio- 
technical  ecosystems. 
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Healthcare  Analytics 

Problem:  Define  healthcare  analytics  from  technical  and 
organizational  perspectives  needed  to  achieve  intelligent  healthcare. 

Healthcare  Infrastructure 


Approach:  Define  an  analytics  framework 


Health  IT  as  an  Ultra 
Large-Scale  System 
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Organizational  Dynamics 
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Adding  Analytics 


Ecosystem 


Organization 


Context: 

-  Culture 

-  Talent/ 
People 


Provides 
yardstick 

Measure  and^ 

support/guide 

improvements 


'  Or 

ganizatior 

Outcomes 

Feedback 
Measure  and 
support/guide 
improvements 


Context: 

-  Standards 

-  Market 
Competitors 

-  Vendors 

-  External 
Regulations 

-  Technology 

-  External 
Research 


Does  Scale  Really  Matter?:  ULS  Systems  Seven  Years  Later 

Software  Engineering  Institute  CarnedeMellon  Linda  Northrop:  May  24,  2013  57 


©2013  Carnegie  Mellon  University 


Best  in  Class  Analytics  Organizations: 
Learning  Organizations 


Architecture 

•  Organizational 
structure 

•  Governance 

•  Enterprise  architecture 

-  Business 
processes 

-  Appiications 

-  Data 

-  Technoiogy 
infrastructure 

-  interoperabiiity 
mechanisms 


I  \C3CCII  L.I  I 
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Best  in  Class  Healthcare  Analytics 
Organizations:  Learning  Organizations 


Influence  ecosystem 


Patient  safety 
and  quality 

Operational 

efficiency 

Financially 

competitive 

Regulatory 

compliant 

Research 

leadership 


Measure  and  shape 

•  Physician  profiling 

•  Training 

•  Technology  change 
management 


Context: 


‘^l5rrect^' 

Predict 


Business  and 
\/lission  Goals 


Provides 
yardstick 

Measure  and 
support/guide 
improvements 


Feedback 
Predict 

A 


Produce 
Predict , 


Organization 

Outcomes 


Work  process  execution 

•  Clinical  care 

•  Administration 

•  Embedded  data  collection 
Internal  research 

•  Medical  science 

•  Clinical  effectiveness 
«  Analytics 


Context: 


I  Explicit  analytics 

Culture  "  Standa 

strategies 

falent/  _J\/!ark©il 

-  Extern* 

Regulati 

-  Technc 

-  Extern* 

Researc 

Architecture 

•  Organizational 
structure 

•  Governance 

•  Enterprise  architecture 

-  Business 
processes 

-  Applications 

-  Data 

-  Technology 
infrastructure 

-  Interoperability 
mechanisms 
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Selected 

UL  S-Sys  terns- Re  I  a  ted 
Research 
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Sample  Published  Work 

(f^  Contextual  Desian 


Contextual  Design 

Collaboration  and  Coordination  in 
Large-Scaie  Socio-technicai 
Ecosystems 

Sociai  Network  Anaiysis 

James  Herbsleb  et  al,  Carnegie  Mellon 
University 


/T 


/T 


Data  Intelligence 
Data  Privacy 
Data  Heterogeneity 


Ecosystem  Modeling 
John  McGregor  et  al,  Clemson  University 


Machine  Learning 
Socio  Linguistics 
Natural  Language  Processing 


Crowdsourcing  Requirements 
Stakeholder  Analysis 


J 

J) 


J 


Self*  computing 

Self-Coordinating  Systems 

Self-Adaptive  Systems 

Dynamic  Adaptive  Systems 

Complex  Adaptive  Systems 

Architecture  Mechanisms  for  Diagnosis  and 

Adaptation 

SEAMS  Community  and  others 

Architecting  ULS  Systems 
End-User  Architecting 
Middleware  for  ULS  Systems 
Domain-Specific  Engineering 
Model-based  Approaches  to  ULS  Systems 
Multi-Product  Lines 
Multi-Sided  Markets 
Data-Intensive  Large-Scale  Systems 
Cloud  Computing  in  the  ULS  Space 

NOTE:  References  at  the  end 
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Domain  Specific  Work 


Climate  Modeling 
NASA  JPL 

Climate  Informatics 

Steve  Easterbrook, 
University  of  Toronto 


Disaster  Management 

Martin  Griss, 
Carnegie  Mellon  Siiicon  Vailey 


Intelligent 

Transportation 

Intel 

University  of  Taiwan 


Health  Information 
Systems 

Kevin  Sullivan, 
University  of  Virginia 


Financial  Markets 

Dave  Cliff, 

University  of  Bristol 


Intelligent  River® 

Clemson  University 


Software  Defect  Analysis  in 
Smart  Grid  Applications 

M.  Ancaari, 

Norwegian  University  of  Science 
and  Technology 
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Selected  SEI  Research 
Targeted  at  ULS  Systems: 
More  Nibbling 
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V, 


High-Confidence  Cyber-Physical 
Systems 


Some  SEI  Research  In  ULS  Systems 


Socio-Adaptive  Systems  Using 
Computationai  Mechanism  Design 
and  Adaptive  QoS 


Edge-Enabled  Tactical 


Systems 


Architecture  in  ULS 
Systems  Context 


Augmented  and  Virtuai  Actors  for 
Threat  Abatement  Readiness 


/f 


Concurrent  Crowdsourcing  of 
Requirements  and  Architectures  for 
Socio-Technical  Infrastructure 
Improvement 


J 
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■  'V 

Architecture  in  ULS  System  Conte^t^ 


\ 


\ 


ULS  system  characteristics  inspire  key  questions  jtoouf 
systems  at  scale.  ' 


'x  _ 


•  What  new  quality  attributes  arise  due  to  scale? 

•  What  types  of  analyses  are  required  to  understand  and  design 
systems  with  these  characteristics? 

•  What  new  architecture  design  principles  needed? 

-  E.g.,  synergy  of  concerns  instead  of  separation  of  concerns? 

•  What  are  the  associated  architectural  tactics,  patterns,  mechanisms? 

•  What  types  of  analyses  and  design  strategies  are  needed  to  design  all 
levels  of  systems  at  scale? 

-E.g.,  population  dynamics,  connectedness/communication 

•  And  what  expertise  is  required  for  this  design? 


Does  Scale  Really  Matter?:  ULS  Systems  Seven  Years  Later 

Software  Engineering  Institute  CarnedeMellon  Linda  Northrop:  May  24,  2013  65 

©  2013  Carnegie  Mellon  University 


Edge-Enabled  Tactical  Systems  (EETS) 

Investigates  architectures  and  technologies  that  adapt  new  generations  of 
mobile  devices  and  sensors  to  support  humans  operating  in  demanding  edge 
environments 


Mobile  technologies  can  enhance  the  manner  in  which 
people  operate  in  tactical  environments 

•  Local  data  caching  with  reach  back  when  available 

•  Cyber-foraging  to  enhance  handheld  and  sensor  device 
capabilities 

•  Flexible  deployment  and  rapid  adaptation  for  new  missions 

•  Context-aware  computing  to  reduce  cognitive  load  and 
conserve  resources 


•  Local,  edge  analytics  to  provide  rapid  data  analysis 

•  Increased  use  of  autonomy  (drones,  robots,  sensors) 

f(  _  ^ 

What  architectures  and  technologies  support  soldiers  and  other  edge  users  in 

customizing  systems  to  unique  needs,  finding  information  that  matters,  and  to 
continue  processing  in  uncertain  computing  environments? 

V  J 
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SEI  and  Broader  Carnegie  Mellon  Collaboration 
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Summary 

#  17  CMU  researchers 

#  21  students 

#  5  SEI  Principal  Investigators 


Software  Engineering  Institute 
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Reflection: 

The  Future  is  Here 
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So,  Where  Are  We? 


The  report  has  been  widely  distributed  via  the  web  and  hard  copy. 
Relevant  research  is  being  conducted  all  over  the  world. 

The  confluence  of  new  technology  is  making  ULS  systems  today. 

•  with  a  profound  impact  on  the  way  society  is  structured  and  how  society 
behaves 

•  substantial  engineering  challenges  are  becoming  widely  recognized  if  still 
poorly  understood. 

-  reliance  on  autonomous  behavior 

-  increased  interaction  and  interdependence  of  socio-technical 
ecosystems 

-  increased  tempo  of  change  across  the  spectrum  of  human  behavior - 
driven  by  human  demand 

There  is  a  wide  range  of  technical  and  non-technical  perspectives  and 
approaches  that  can  be  brought  to  bear. 
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ClimatG  Change  (term  used  by  David  West  at  Code  Freeze  2013) 

Characteristics  of  ULS  systems  arise  because  of  their  scale. 

•  Decentralization 

•  Inherently  conflicting,  unknowable,  and  diverse 
requirements 

•  Continuous  evolution  and  deployment 

•  Heterogeneous,  inconsistent,  and  changing  elements 

•  Erosion  of  the  people/system  boundary 

•  Normal  failures 

•  New  paradigms  for  acquisition  and  policy 

These  are  real. 
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Opportunities  and  Threats 


Opportunity 


Threat 
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Implications  for  How  We  Do  Our  Work 


SYSTEMS  MUST  BE:  Responsible  | 

Responsive  |  Adaptive  -  SO  MUST  WE 

Software  Engineering  Institute 

Does  Scale  Really  Matter?:  ULS  Systems  Seven  Years  Later 

Carnegie  Mellon  Linda  Northrop:  May  24,  2013  72 

©2013  Carnegie  Meiion  University 

Putting  Technology  to  Work:  a  Few  Take  Aways 


Context  is  key 

•  Computation  needs  to  be  organic  with  human  incentives 
and  human  workflow 

•  Technology  standards  do  not  ensure  interoperability 

•  Multi-disciplinary  approach  is  essential 

Big  data  and  machine  learning  don’t  help  without  an 
analytics  framework,  feedback  loops,  and  analytics-driven 
sensing 

Humans,  computational,  and  autonomous  entities  are  peers 
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Research  Progress  -  My  Assessment 

^6.1  Human  Interaction 
^6.2  Computational  Emergence 
^6.3  Design 

^6.4  Computational  Engineering 
^ 6.5  Adaptive  System  Infrastructure 
^ 6.6  Adaptable  and  Predictable  System  Quality 
^6.7  Policy,  Acquisition,  and  Management 


Progress  has  been  made  on  all  these  fronts  and  others. 

And  yet... there  is  a  fast  growing  gap  between  our  research  and 
^  reality. _ 
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What  I  See 


Machine  Learning 


Computer  Science 
Socioiogy 

Business 


Rhetoric 


Organizationai  Dynamics 
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Summing  It  Up 


Scale 

M  a  tte  rs 


=^=-  Software  Engineering  Institute 


•  ULS  systems  are  in  our  midst  and 
the  changes  to  our  social  fabric 
and  institutions  are  significant. 

•  In  hindsight,  we  were  probably  too 
conservative  in  our  report. 

•  Recent  technologies  have 
exacerbated  the  pace  of  scale 
growth  -  allowing  us  to  transcend 
time  and  space. 

•  There  are  great  opportunities. 
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Food  for  Thought 


•  Is  our  research  a  match? 

•  Do  we  have  the  right 
incentives  and  mindset  for  the 
needed  multi-disciplinary 
approach? 

•  Will  we,  the  software 
engineering  research 
community,  make  a 
difference? 
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Thanks  To  The  Entire  ULS  System  Study  Team 


Does  Scale  Really  Matter?:  ULS  Systems  Seven  Years  Later 

Software  Engineering  Institute  CarnedeMellon  Linda  Northrop:  May  24,  2013  79 

©  2013  Carnegie  Mellon  University 


Thanks  To  My  SEI  and  Campus  Colleagues 
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A  Special  Thank  You  and  Tribute 
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Interaction 
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Contact  Information 


Linda  Northrop 

SEI  FgIIow 
Chief  Scientist 
Software  Solutions  Division 
Telephone:  412-268-7638 
Email:  lmn@sei.cmu.edu 
Website:  http://www.sei.cmu.edu/uls/ 


U.S.  Mail: 

Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213-3890 


SEI  Fax:  412-268-5758 
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